Repository navigation
PLT-7830: graph-node: subgraph runner exits when the DB is still down at restart time (store.restart / health errors) - #6
Conversation
…estart (PLT-7830) A store error raised while handling a non-deterministic error, while unfailing, or by restart() after the first block stream started no longer ends the runner: health()/fail_subgraph()/unfail errors are logged, a failed restart backs off with the runner's backoff and retries, and an unreadable chain head processes the block instead of ending the runner. A failure to record a deterministic error backs off and retries so the error gets recorded. Retries happen only once a block stream registered the cancel guard, so an unassigned runner stops within one backoff interval; Initializing and the first Restarting keep returning the error. Tests in tests/tests/writer_restart.rs put subgraphs.deployment or subgraphs.head behind a view that fails every read from the injected write failure until the outage ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: zerion-hive-worker[bot] <318250561+zerion-hive-worker[bot]@users.noreply.github.com>
|
Reviewer: reviewer-PLT-7830 @ 65c8afa Review: PLT-7830Outcome: CHANGES REQUESTED. One blocker, in the deterministic arm. The other paths are correct, and CI is green at this head. Blocker 1: the deterministic-arm retry resumes past the failing block (
|
…ing past its block (PLT-7830) When fail_subgraph(.., true) fails, the deterministic arm retried with Action::Restart. The fatal block's PoI-only batch is already written and a healthy writer's restart() is a no-op, so the stream resumed past the block and the error was never recorded. Retry fail_subgraph in place with the backoff instead, and stop when the runner is canceled. The new test fails the fatal-error status update until the outage ends and asserts the error is recorded at block 3 and the head never passes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: zerion-hive-worker[bot] <318250561+zerion-hive-worker[bot]@users.noreply.github.com>
|
Reviewer: reviewer-PLT-7830 @ 23a22e1 Review: PLT-7830 (re-review; changes since 65c8afa)Outcome: APPROVE. The fix resolves Blocker 1 from attempt 1 ( Blocker 1: resolved
Acceptance Criteria
Verdict: APPROVENotes (non-blocking, all
|
|
Deployed: v0.43.0-zerion3 (tag at 1e572ab, ECR sha256:aa584d57e13222aa7abf6d49f8828c0b4e754e80d98da63b0fda6244f04dbdce) · image.tag k8s-cluster eksctl 6c8e1bfa5885062b86b69bb934de18a88b7ad99f · production |
PLT-7830: graph-node: subgraph runner exits when the DB is still down at restart time
Ports the shape of upstream graphprotocol#6730 (closed unmerged; issue graphprotocol#6741, paths 1 and 3) onto the
zerion/v0.43.0patch line, without graphprotocol#6730'sstore.restart()change, which this branch already has. It adds the unassign guard that graphprotocol#6730 does not have.Changes
All in
core/src/subgraph/runner/mod.rs:handle_action,let already_failed): a store error fromhealth()orfail_subgraph()is logged atwarninstead of being returned. The arm still backs off and returnsAction::Restart. Ifhealth()fails, the error is still recorded withfail_subgraph().handle_action): a store error fromunfail_non_deterministic_error()is logged atwarnand treated asBehindErrorBlock, so the next block tries again.handle_action): whenfail_subgraph(.., true)fails, the runner logs aterror, backs off and retriesfail_subgraphin place. It does not restart: the fatal block's PoI-only batch is already written, so a restart would resume past the block. Between attempts it checksis_canceled(), and a canceled runner returnsAction::Stop, whichprocess_block_stateturns intocancel(). Once the error is recorded, the deterministic error ends the runner as before. I chose this over "log and return" because an unrecorded deterministic error leaves the deploymenthealthyand stopped. (This changed in review attempt 1. The first version restarted, and that resumed past the fatal block.)restart()failures (run_inner,RunnerState::Restarting, plus the newback_off_before_restart): anUnknownerror fromrestart()(store.restart()orstart_block_stream()) backs off withself.state.backoffand retriesRestarting { reason }. This happens only whencan_observe_unassign()is true, meaningcancel_handleisSome.is_canceled()is checked before and after the sleep, and a canceled runner goes through the existingcancel(), which returnsStopped { Unassigned }orDuplicate.process_block_state→ newis_close_to_chain_head): an error fromchain_head_ptr()is logged atwarnand the block is processed rather than skipped. The runner no longer ends on it.tests/tests/writer_restart.rs: there are three new tests, serialised by the file's existingONE_TEST_AT_A_TIMEmutex. AnOutagehelper renamessubgraphs.deploymentorsubgraphs.headand puts a view with the old name in its place. Every statement that reads the view fails withinjected outage: database unavailablefrom the moment the existing injected write failure (fail_first_write_reaching) fires untilOutage::end. A sequence counts the failed reads, andDroprestores the table even when a test panics.fail_first_write_reachingnow returns the deployment namespace.Acceptance Criteria
health(), fromfail_subgraph()in the non-deterministic arm and fromunfail_non_deterministic_error()no longer end the runner. Tested bystore_errors_while_handling_a_non_deterministic_error_keep_the_runner, with the outage onsubgraphs.deployment. The test waits for 2 failed reads (health+fail_subgraph), ends the outage, and asserts that the head reaches block 5, past the failing block 2, with no fatal error. Runs are below. Theunfailpath has no dedicated test; its change is the logged fallback in the success arm.restart()store error after the first block stream backs off and retries. Tested bystore_errors_at_restart_back_off_and_retry, with the outage onsubgraphs.head, whichstore.restart()reads when it builds the new writer.health/fail_subgraphreadsubgraphs.deployment, so they succeed. The test waits for 2 failedstore.restart()reads, asserts the head behind the outage is still 1, ends the outage, and asserts the head reaches 5 with no fatal error.grep -n '\.await?' core/src/subgraph/runner/mod.rshit on the branch head.unassigned_runner_stops_while_restart_backs_offuses thesubgraphs.headoutage and never ends it. It waits for 2 failed restarts, which proves the runner is still in the retry loop, then callsinstances.remove(&deployment.id)(the instance manager'sstop_subgraphpath) and asserts that the runner future completes within 3 s (backoff ceiling 1 s plus jitter) and not with the injected-outage error. The runner log showsTerminating the subgraph runner because subgraph was unassignedand thenStopping subgraph - unassigned.Initializingand the firstRestartingare unchanged:RunnerState::Initializing => self.initialize().await?(line 516) still returns the error. In the firstRestarting,cancel_handleisNone, so the guardif self.can_observe_unassign()(line 527) fails andres => res?(line 530) returns the error as before. No cancel guard is registered early.SubgraphRunnerError::Duplicateis never retried: onlyUnknownmatches, andcancel()still returnsDuplicate. Deterministic errors still end the runner once they are recorded, and a failed recording never moves past the fatal block. Tested bydeterministic_error_is_recorded_without_passing_its_block: a trigger fails only the fatal-error status update onsubgraphs.deployment, and thefatal-errorsubgraph fails at block 3. The test waits for 2 failed recordings and asserts the head is ≤ 3. It then ends the outage and asserts the runner returnsErr, the head is ≤ 3, andfatal_erroris deterministic at block 3. At 65c8afa it failed in each of 3 runs (the runner ended during the outage: Ok(()): the runner resumed past block 3). On 23a22e1 it passed in each of 3 runs. Locally,cargo test -p graph-tests --test runner_tests -- --test-threads=2gave 16 passed, andcargo test -p graph-corepassed. CI'sRun unit testsandSubgraph Runner integration testschecks are the gate.v0.43.0-zerion3and track the rollout.AC3:
.await?dispositions (branch head)new_block_stream(..)instart_block_streamrestart(), covered by theRestartingback-off once a stream has startedtriggers_adapter.parent_ptr()ininitializeInitializing, where unassign is not observable (AC4)store.unfail_deterministic_error()ininitializeInitializing, where unassign is not observable (AC4)store.flush()ininitialize(max end block reached)Initializing, where unassign is not observable (AC4)store.restart()inrestartRestarting(AC4)start_block_stream()inrestartRestarting(AC4)store.flush()infinalizeself.initialize()inrun_innerInitializing, where unassign is not observable (AC4)back_off_before_restart()inrun_innerDuplicatefromcancel()await_block()inrun_innerDuplicatefromcancel()process_block_state()inrun_innerDuplicate; its store and chain-store errors are now handled inside (lines 595, 612)handle_action()inprocess_block_stateErris the deterministic error after it was recordedmaybe_cancel()intransact_block_stateProcessingErrorinsideprocess_block;handle_actionclassifies it (non-deterministic → retry)refetch_block()inprocess_blockProcessingError, classified byhandle_actionprocess_dynamic_data_sources()inprocess_blockProcessingError, classified byhandle_actionprocess_offchain_triggers()inprocess_blockProcessingError, classified byhandle_actiontransact_block_state()inprocess_blockProcessingError, classified byhandle_action(the injected write failure goes this way)chain.chain_head_ptr()inis_caught_up.non_deterministic()?, so it is retried by the non-deterministic armentity_cache.as_modifications()inhandle_offchain_triggers.non_deterministic()?store_poi_entity()inupdate_proof_of_indexing.non_deterministic()?These
.await?sites are no longer in the grep:chain_head_ptr()inprocess_block_state(path 3, formerly ~:566) is now logged, throughis_close_to_chain_head.unfail_non_deterministic_error()is logged.fail_subgraph(.., true)is retried in place (in the deterministic arm, without a restart).health()andfail_subgraph(.., false)are logged.Reproduction runs
I ran the tests locally against Postgres 16 and IPFS in Docker. At base, the runner fix was reverted in the worktree (
git checkout -- core/src/subgraph/runner/mod.rs, so the runner code isa1a67f93) and the new tests were kept.At base
a1a67f93, three runs gave the same result. Each run printed:On the branch, three runs of the new tests plus one run of the whole file all passed:
Branch runner log, per test:
Failed to read subgraph health×1,Failed to record non-deterministic error×1, thenStop block reached.Subgraph failed to restart, retrying×2, thenStop block reached.Subgraph failed to restart, retrying×2, thenTerminating the subgraph runner because subgraph was unassigned.Departures
tests/tests/writer_restart.rs, not a new file. They share its serialisation and write-failure injection, and CI already runs that file (just test-writer-restartin the runner-test job), sojustfileandci.ymlneed no change.health()failure: the error is still recorded withfail_subgraph(), as core: Keep the subgraph runner alive through transient store errors graphprotocol/graph-node#6730 does, rather than skipping the record.runner_testsrun, at default parallelism, failed 6 tests during database setup (database setup failed: database query timed outinpool/coordinator.rs, then cascadingPoisonErrors) against my local Docker Postgres. A rerun with--test-threads=2passed all 16. I count this as local environment, not this change. CI is the gate.cargo check --releaselocally. CI'sBuild in release modecovers it.Testing
cargo fmt --all: clean.cargo clippy -p graph-core -p graph-tests --all-targets: no warnings.tests/tests/writer_restart.rs:store_errors_while_handling_a_non_deterministic_error_keep_the_runner,store_errors_at_restart_back_off_and_retry,unassigned_runner_stops_while_restart_backs_off,deterministic_error_is_recorded_without_passing_its_block.Resolves PLT-7830
🤖 Generated with Claude Code
Produced by a hive session (PLT-7830)